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ABSTRACT 

A description of the evaluation plan used in the 
Reading Improvement Project in Cleveland's target Title I elementary 
schools is presented. The plan's formation was influenced by two 
guidelines: (1) the necessity of providing for procedures to overcome 

threats to the validity of the findings so that some sound assessment 
of program effects could be made and (2) a desire to remain sensitive 
and responsive to realities of the school worlds 

involved--particularly concerns of pupils and staff participating in 
the project. Three of the critical issues encountered, among many, 
and the solutions found for them are given. The issues were (1) 
random assignment of pupils to project services, (2) choice of a 
model of analysis appropriate to the program's data collection 
procedures, and (3) the determination of whether or not experimental 
and control pupils received significantly different final reading 
marks from their classroom teachers. Covariates and dependent 
variables used as factors in the evaluation are listed. (Author/NH) 
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Evaluation of the Reading Improvement Project in Cleveland's 
target Title I elementary schools might be described as having evolved 
through a unique combination of circumstance and deliberate design. Two 
guidelines influenced formulation of the evaluation plan — first, the 
necessity to provide for procedures to overcome threats to validity of 
the findings so that some sound assessment of program effects could be 
-<a de; and second, a desire to remain sensitive and responsive to realities 
of the school worlds involved — particularly concerns of pupils and staff 
participating in the project. 

This discussion centers on three from among several critical 
issues encountered. It reports the solutions attempted in structuring a 
sound evaluation plan to operate within the setting of the inner-city 
public school. 

The first issue focused on random assignment of pupils to 
project services. 

The problem was of concern because evaluation standards 

# 

necessitated control, and yet the intent of Title I legislation, as you 
know, as well as our own commitment in Cleveland as educators, would not 
permit denial of service to youngsters in need, just to satisfy an 
evaluation plan. It was recognized that public schools do not exist 
primarily for the support and proliferation of evaluation schemes. 

Fate, perhaps, had a hand because identification of pupils 
according to program criteria produced a much larger population than 
could be accommodated by program resources. Approximately 800 places 



were available, and almost 1,600 pupils met the criteria in the primary 
grades throughout the 21 target schools.. 

Random assignment, therefore, provided a fairer solution than 
otherwise would have been possible for the question of who shall be 
served when so many have been identified. It ensured as well an experimental- 
control framework for assessment of effects. 

Random assignment had another important contribution to the 
program. The schools involved in the project have mobility rates ranging 
from about 25 to almost 100 per cent. Pupils in these schools can be 
anticipated to move in and out during the course of a program. Random 
assignment was a means for providing for systematic replacesient as places 
would become available in service groups. 

This replacement was done at four entry points during the year. 

It helped to clarify for school and program administrators the issue of 
relationship between term of program service and level of improvement in 
reading performance. 

For example, grade 3 pupils assigned in September gained an 
average of 1.1 grade equivalent units in comprehension. This gain was 
twice the average gain observed for pupils assigned in the February 
period, which was .54 grade equivalent units. 

A second issue involved choice of a model of analysis appropriate 
to the program's data collection procedures — now that random assignment 
could be employed. Our questions of interest required measurement 
of an array of variables related to reading performance. It was 
important, for example, that not only scores on standardized reading 
tests of vocabulary and comprehension be compared for experimental and control 

groups, but that observations be assembled about the relationships between 
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attendance and -poverty level of pupils and their reading performance. 

Another critical area of interest was the determination of 
whether or not experimental and control pupils received significantly 
different final reading marks from their classroom teachers. It was 
also considered important that consideration be given in the evaluation 
plan to differences in the classroom teachers* ratings of experimental 
and control pupils* performance in classroom reading groups, their 
confidence, adjustment to school, attitude toward school, and attitude 
toward reading. 

As can be seen from this slide, from the evaluation needs, 
there emerged three variables which would function as co-variates— 
scholastic apitude test score, poverty index, and attendance. 

Slide 1 here 

Ten variables emerged as the dependent factors in the evaluation 
plan. This slide indicates these variables included: Vocabulary test 

score, comprehension test score, final mark in reading, ratings of class- 
room reading performance, work in reading group, written work, pupil 
confidence, school adjustment, attitude toward school and attitude toward 
reading • 

Slide 2 here 

«««»»»»»»•>« • • • * 

This data just described were cast into a 2 x 3 x 2 model 
involving the factors: sex, grade, and experimental-control. This slide 

shows the model with a total of 252 cases allotted disproprotionately 
across 12 subclasses. 

Slide 3 here 



It was apparent that an appropriate analysis procedure for 
this evaluation would have to take into account the dependencies existing 
between the variables described. It was also necessary to provide a 
solution for disproportionate subclass numbers which occur with frequency 
in data from the real world. Finally, it should be a technique which 
could deal with correlation between the variables involved. 

It was also apparent that univariate analysis procedures 
could not deal with correlations between all of the variables involved 
in this program; nor could these procedures produce independent statistical 
tests. Multivariate analysis of covariance was selected, therefore, as 
the appropriate procedure. 

The multivariate approach would use a single probability 
statement applicable to all variables jointly and would be based upon a 
known sampling distribution from which required probabilities could be 
obtained. It would allow for inspection of the differences betv/een 
treatment effects, so that the direction and relative size of effect on 
each of the dependent variables could be determined. After test of main 
effects of variables under this procedure, step-down tests would permit 
investigation of dependent variables in an ordering chosed by the 
investigator. 

Results of the multivariate test indicated that the Reading 
Improvement Program produced a significant upgrading of reading performance 
for experimental children. In all grades, experimental children reflected 
superior reading performance to that observed for their controls in terms 
of vocabulary and comprehension test scores. 

For boy and girl contrasts, the multivariate test indicated no 
differences for any of the dependent variables. 
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In terms of grade level contrasts, the multivariate tests 

revealed that the greatest impact of the program took place at grade 3 

* 

where reading comprehension performance, school adjustment and final reading 
mark were significantly higher for grade 3 experimental pupils than all 
other pupils. 

Teachers were shown to have given significantly higher 
classroom reading performance ratings to grade 1 pupils, while grade 2 
pupils reflected highest reading attitude ratings of all the grade levels. 

It was found, also, that final marks in reading given by the 

classroom teachers were negatively correlated with performance on the 

« 

vocabulary and comprehension tests. Other ratings given by the class- 
room teachers to both control and experimental children were also found 
to be inversely related to results on the standardized tests. Correlations 
between the teachers* ratings spread across a wide range with a top 
positive correlation of .58 to a low negative correlation of .11. It 
had been anticipated that there would be a positive relationship between 
teacher ratings in these areas rather than the wide range observed. This 
finding suggests in-service needs for teachers in identifying appropriate 
and inappropriate pupil behaviors. 

A third issue centered on the structuring of program goals so 
that expectations of what pupils should and could do because of treatment 

would be more specifically identified. Establishment of individualized 

, € 

goals came through the use of a reading formula of expectancy, the Bond- 
Tinker method. 

Slide 4 here 

As can be seen from the slide, the Bond-Tinker formula generates 
a reading expectancy expressed in grade equivalent units by using the 
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product obtained from multiplying the pupil’s years in school (beyond 
kindergarten, that is) by the I.Q, score divided by 100, to which is 
added a constant of 1.00 to reflect the pupil’s grade equivalent score 
of 1.0 when he begins the first grade. 

Typical goals for reading programs usually state something 
to the effect that children served in general will gain a year in 
achievement as measured by standardized reading tests. Such a gross 
goal was considered inappropriate and insensitive to what the Reading 
Program was attempting to accomplish. Individualized goals which 
established each pupil’s expectancy took into account the present condition 
of the pupil as well as what would be the probable development of his 
reading skill. 

It has been demonstrated that the Bond-Tinker method identifies 
greater discrepancies between reading level and expectancy for under- 
achieving pupils who fall in the below-average range of scholastic ability. 
This tendency of the formula was considered appropriate because expectations 
should be raised for this group of pupils who were probably actually 
performing in such a state because of their reading problems. 

The final step in individualizing goals included the addition of 
a criterion. The criterion stipulated that a dimension of tolerable 
performance would be allowed. It would be within a half-year of the 
pupil's reading expectancy, using a grade placement score. This was 
considered an appropriate performance interval for the primary grade pupil. 

The goal for each pupil then became: 

To improve reading performance of educationally deprived 
pupils by bringing them up to a level appropriate for their 
reading expectancy which shall be determined by the Bond-Tinker 
formula; the appropriate level being described as within one 
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half-year of the reading expectancy in terras of grade 
equivalent score units. 

The use of a reading expectancy for pupils provided an 
additional evaluation dimension for assessing projects effects by 
facilitating an individual-vs-self comparison within an overall plan 
of experimental-control contrasts. Frequently, reading improvement 
programs look only to the advantage anticipated in experimental -control 
contrasts rather than a concern that pupils served actually will 
perform at an improved level geared to their probable reading growth. 

As can be seen from this slide, grade 2 pupils moved upward 
toward their reading expectancies. Forty per cent attained the criterion 
level of within at least a half-year of their expectancies, while another 
38 per cent fell within a year to .6 of a year of their expectancies. 

Fifty per cent had began the program 1.1 of year to 1.5 year below their 
expectancies. 

Slide 5 here 

In summary, this approach to an evaluation of reading project 
in the inner city school provided hopefully a sensitive heart to rigorous 
procedures, nonetheless overcoming the threats to valid inference about 
the effects of the program. Not to be overlooked, however, is the fact 
that contrary to the findings of Title I programs, which have been put 
with one fell swoop by some commentators into ineffective expenditures of 
time and money, the Reading Improvement Program appears to have accomplished 
what is said it would. 

Paper presented in AERA Symposium - March 5, 1970 
Minneapolis, Minnesota 
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